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Abstract 

We consider an untmsted server storing shared data on behalf of clients. We show that no storage 
access protocol can on the one hand preserve sequential consistency and wait-freedom when the 
server is correct, and on the other hand always preserve fork sequential consistency. 

1 Introduction 

We examine an online collaboration facility providing storage and data sharing functions for remote 
clients that do not communicate directly GHHEElEl]. Specifically, we consider a server that implements 
single-writer multi-reader registers. The storage server may be faulty, potentially exhibiting Byzantine 
faults iflOl l8l ITT1 l2l. When the server is correct, strong liveness, namely wait-freedom [5], should be 
guaranteed, as a client editing a document does not want to be dependent on another client, which could 
even be in a different timezone lfl4l . In addition, although read/write operations of different clients 
may occur concurrently, consistency of the shared data should be provided. Specifically, we consider a 
service that, when the server is correct, provides sequential consistency, which ensures that clients have 
the same view of the order of read/write operations, which also respects the local order of operations 
occurring at each client Q ■ Sequential consistency provides clients with a convenient abstraction of a 
shared storage space. It allows for more efficient implementations than stronger consistency conditions 
such as linearizability especially when the system is not synchronized (T). 

In executions where the server is faulty, liveness obviously cannot be guaranteed. Moreover, with a 
Byzantine server, ensuring sequential consistency is also impossible [2]. Still, it is possible to guarantee 
weaker semantics, in particular so-called forking consistency notions El El. These ensure that when- 
ever the server causes the views of two clients to differ in a single operation, the two clients never again 
see each other's updates after that. In other words, if an operation appears in the views of two clients, 
these views are identical up to this operation. 

Originally, fork-linearizability was considered JUQI1I2. In this paper, we examine the weaker fork 
sequential consistency condition, recently introduced by Oprea and Reiter |[TT1 . who showed that this 
new condition is sufficient for certain applications. However, to date, no fork-sequentially-consistent 
storage protocol has been proposed. In fact, Oprea and Reiter suggested this as a future research direc- 
tion ifTTI . Furthermore, Cachin et al. [2] showed that the stronger notion of fork-linearizability does not 
allow for wait-free implementations, but conjectured that such implementations might be possible with 
fork sequential consistency. Surprisingly, we prove here that no storage access protocol can provide fork 
sequential consistency at all times and also be sequentially consistent and wait-free whenever the server 
is correct. This generalizes the impossibility result of Cachin et al. O, and requires a more elaborate 
proof. 
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In this paper we require only sequentially consistent semantics when the server is correct. Though 
one may also consider stronger semantics, such as linearizability, for this case, as our goal is to prove an 
impossibility result, it suffices to address sequential consistency. Our impossibility result a fortiori rules 
out the existence of protocols with stronger semantics as well. 

2 Definitions 

System model. We consider an asynchronous distributed system consisting of n clients Ci, . . . , C n , 
a server S, and asynchronous FIFO reliable channels between the clients and S (there is no direct 
communication between clients). The clients and the server are collectively called parties. System 
components are modeled as deterministic I/O Automata Q. An automaton has a state, which changes 
according to transitions that are triggered by actions. A protocol P specifies the behaviors of all parties. 
An execution of P is a sequence of alternating states and actions, such that state transitions occur 
according to the specification of system components. 

All clients follow the protocol, and any number of clients can fail by crashing. The server might be 
faulty and deviate arbitrarily from the protocol, exhibiting so-called "Byzantine" faults [12J. A party 
that does not fail in an execution is correct. The protocol emulates a shared functionality F to the clients, 
defined analogously to shared-memory objects. 

Events, operations, and histories. Clients interact with the functionality F via operations provided 
by F. As operations take time, they are represented by two events occurring at the client, an invocation 
and a response. An operation is complete if it has a response. For a sequence of events a, complete(a) 
is the maximal subsequence of a consisting only of complete operations. 

A history is a sequence of requests and responses of F occurring in an execution. An operation o 
precedes another operation d in a sequence of events a, denoted o < a d, whenever o completes before 
d is invoked in a. Two operations are concurrent if neither one of them precedes the other. A sequence 
of events is sequential if it does not contain concurrent operations. A sequence of events it preserves 
the real-time order of a history a if for every two operations o and d in ir, if o < cr d then o < w d . For 
a sequence of events n, the subsequence of it consisting of events occurring at client d is denoted by 
n\d- For a sequential it, the prefix of it ending with operation o is denoted by tt°. 

An execution is admissible if the following two conditions hold: (1) the sequence of events at each 
client consists of alternating invocations and matching responses, starting with an invocation; and (2) the 
execution is fair. Fairness means, informally, that the execution does not halt prematurely when there 
are still steps to be taken or messages to be delivered (we refer to the standard literature for a formal 
definition of admissibility and fairness Il9ll). 

Read/write registers. A functionality F is defined via a sequential specification, which indicates the 
behavior of F in sequential executions. 

The basic functionality we consider is a read/write register X. A register stores a value v from a 
domain X and offers read and write operations. Initially, a register holds a special value _L X . When a 
client Ci invokes a read operation, the register responds with a value v, denoted readi(X) — ► v. When Cj 
invokes a write operation with value v, denoted writei(X, v), the response of X is an acknowledgment, 
denoted by OK. The sequential specification requires that each read operation from X return the value 
written by the most recent preceding write operation, if there is one, and the initial value otherwise. We 
assume that the values written to every particular register are unique, i.e., no value is written more than 
once. This can easily be implemented by including the identity of the writer and a sequence number 
together with the stored value. 
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In this paper, we consider single-writer/multi-reader (SWMR) registers, where for every register, 
only a designated writer may invoke the write operation, but any client may invoke the read operation. 

Sequential consistency. One of the most important consistency conditions for concurrent access is 
sequential consistency [7], which preserves the real-time order only for operations by the same client. 
This is in contrast to linearizability, which must preserve the real-time order for all operations. 

Definition 1 (Sequential consistency 0). A history a is sequentially consistent w.r.t. a functionality 
F if it can be extended (by appending zero or more response events) to a history a', and there exists a 
sequential permutation it of complete(a') such that: 

1. For every client d, the sequence 7c\d preserves the real-time order of a; and 

2. The operations of 7r satisfy the sequential specification of F. 

Intuitively, sequential consistency requires that every operation takes effect at some point and occurs 
somewhere in the permutation n. This guarantees that every write operation is eventually seen by all 
clients. In other words, if an operation writes v to a register X, there cannot be an infinite number of 
subsequent read operations from register X that return a value written to X prior to v. 

Wait-freedom. A shared functionality needs to ensure liveness. A common requirement is that clients 
are able to make progress independently of the actions or failures of other clients. A notion that formally 
captures this idea is wait-freedom . 

Definition 2 (Wait-free history). A history a is wait-free if every operation by a correct client in a is 
complete. 

Fork sequential consistency. The notion of fork sequential consistency ifTTTl requires, informally, 
that when an operation is observed directly or indirectly by multiple clients, then the history of events 
occurring before the operation is the same at these clients. For instance, when a client reads a value 
written by another client, the reader is assured to be consistent with the writer up to its write operation. 

Definition 3 (Fork sequential consistency). A history a is fork-sequentially-consistent w.r.t. a func- 
tionality F if it can be extended (by appending zero or more response events) to a history a', such that 
for each client C,, there exists a subsequence dj of complete(a') and a sequential permutation -K{ of a, 
such that: 

1. All complete operations in a\d are contained in o~i, 

2. For every client Cj, the sequence %i\cj preserves the real-time order of a; 

3. The operations of 7Tj satisfy the sequential specification of F; and 

4. (No-join) For every o G 7Tj Pi TTj, it holds that n° = 7r? . 

A permutation ir-i satisfying these properties is called a view of C{. 

Note that a view 7Tj of Cj contains at least all those operations that either occur at C, or are apparent 
from Ci 's interaction with F. A fork-sequentially-consistent history in which some permutation tt of 
complete (a 1 ) is a possible view of all clients is sequentially consistent. 

We are now ready to define a fork-sequentially-consistent storage service. It should guarantee se- 
quential consistency and wait-freedom when the server is correct, and fork sequential consistency oth- 
erwise. 
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Definition 4 (Wait-free fork-sequentially-consistent Byzantine emulation). A protocol P is a wait- 
free fork-sequentially-consistent Byzantine emulation of a functionality F on a Byzantine server S if P 
satisfies the following conditions: 

1 . If S is correct, the history of every admissible execution of P is sequentially consistent w.r.t. F 
and wait-free; and 

2. The history of every admissible execution of P is fork sequentially consistent w.r.t. F. 

We show next that wait-free fork-sequentially-consistent Byzantine emulations of SWMR registers 
are impossible. 

3 Impossibility of Wait-Freedom with Fork Sequential Consistency 

Theorem 1. There is no wait-free fork-sequentially-consistent Byzantine emulation of n > 2 SWMR 
registers on a Byzantine server S. 

Proof. Towards a contradiction assume that there exists such a protocol P. Then in any admissible 
execution of P with a correct server, every operation of a correct client completes. We next construct 
three executions a, f3, and 7 of P, shown in Figures [l]-[3] All three executions are admissible, since 
clients issue operations sequentially, and every message sent between two correct parties is eventually 
delivered. There are two clients C\ and C2, which are always correct, and access two SWMR registers 
X\ and X2. Protocol P describes the asynchronous interaction of the clients with S; this interaction is 
depicted in the figures only when necessary. 

Execution a. In execution a, the server is correct. The execution is shown in Figure [T] and begins 
with four operations by C2: first C2 executes a write operation with value v\ to register X2, denoted w\, 
then an operation reading register X%, denoted r\, then an operation writing v-i to X2, denoted w\, and 
finally again a read operation of X\, denoted r\. Since S and C2 are correct and P is wait- free with a 
correct server, all operations of C2 eventually complete. 

write^X^u) 

Ci k k 1 1 ► 




■ r 2 zl read^XJ-^u 
C 2 H 1 I I I II " I I I \k H ft " HH HI 




to 



Figure 1 : Execution a, where S is correct. 

Execution a continues as follows. C\ starts to execute a single write operation with value u to X\, 
denoted w\. Every time a message is sent from C\ to S during this operation, and as long as no read 
operation by C2 from X\ returns a value different from _L, the following steps are repeated in order, for 
* = 3, 4, . . . : 

(a) The message from C\ is delayed by the asynchronous network; 

(b) C2 executes an operation writing Vi to X2, denoted w\; 
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(c) Ci executes an operation reading X\ , denoted r\ ; and 

(d) the delayed message from C\ is delivered to S. 

Note that w\ and r\ complete by the assumptions that P is wait-free and that S is correct. For the same 
reason, operation w\ eventually completes. After w\ completes, and while Ci does not read any non-_L 
value from X\, C2 continues to execute alternating operations w\ and r|, writing V{ to X2 and reading 
X\, respectively. This continues until some read returns a non-_L value. Because S is correct, eventually 
some read of X\ is guaranteed to return u 7^ _L by sequential consistency of the execution. We denote 
the first such read by rf. This is the last operation of C2 in a. If messages are sent from C\ to S after 
the completion of rf, they are not delayed. 

Note that the prefix of a up to the completion of rf is indistinguishable to C2 from an execution in 
which no client writes to X\, and therefore r\, r\, and rf return the initial value _L. Hence, z > 4. 

We denote the point of invocation of w^ -1 in a by to- It is marked by a dotted line. Executions (3 
and 7 constructed below are identical to a before to, but differ from a starting at to- 




read^XHv^ 
I I ► 



Figure 2: Execution (3, where S is correct. 



Execution 0. We next define execution f3, shown in Figure [2j in which the server is also correct. 
Execution (3 is identical to a up to the end of r^ -2 (before to), but then C2 halts. In other words, 
the last two write-read pairs of C2 in a are missing in f3. Operation w\ is invoked in j3 like in a 
and begins after the completion of r\ (notice that r\ is in f3 since z > 4). Because the protocol is 
wait-free with the correct server, operation w\ completes. Afterwards, C\ repeatedly reads X2 until 
v z -2 is returned. Because the execution is sequentially consistent with the correct server, a read of 
X2 eventually returns v z -2- We denote the i-th read operation of C\ by r\ and the read operation that 
returns v z -2 by r\. 

Execution 7. The third execution 7 is shown in Figure [3j here, the server is faulty. Execution 7 
proceeds just like the common prefix of a and before to, and client C\ invokes w\ in the same way 
as in a and in (3. From to onward, the server simulates (3 to C\ . This is easy because S simply hides 
from C\ all operations of C2 starting with ■ The server also simulates a to C2. We next explain 
how this is done. Notice that in a, the server receives at most one message from C\ between to and the 
completion of rf , and this message is sent before to by construction of a. If such a message exists in a, 
then in 7, which is identical to a before to, the same message is sent by C\. Therefore, the server has 
all information needed to simulate a to C2 and rf returns u. 

Thus, 7 is indistinguishable from a to C2 and indistinguishable from f3 to C\. However, we next 
show that 7 is not fork-sequentially-consistent. Consider the sequential permutation 1T2 required by the 
definition of fork sequential consistency, i.e., the view of C2. As the real-time order of C2's operations 
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Figure 3: Execution 7, where S is faulty and simulates a to C2 and (3 to C\. 



and the sequential specification of the registers must be preserved in 712, and since r\, r^p return ± 
but r| returns u, we conclude that w\ must appear in tt2 and is located after r%~ but before rf . Because 
is one of Cx's operations, it also appears in 7Ti. By the no-join property, the sequence of operations 
preceding w\ in TT2 must be the same as the sequence preceding w\ in 7Q. In particular, w 7 ^ and w z 2 ~ 2 
appear in 7Ti before w\, and wf 2 precedes w^ 1 ■ Since the real-time order of Ci's operations must be 
preserved in 7Ti, operation u>i and, hence, also , appears in tt\ before r[. But since iu| writes 
u 2 _i to X2 and reads t> 2 _2 from X2, this violates the sequential specification of X2 iy z -2 is written 
only by ti^ -2 )- This contradicts the assumption that P guarantees fork sequential consistency in all 
executions. □ 



4 Conclusions 



When clients store their data on an untrusted server, strong guarantees should be provided whenever 
the server is correct, and forking conditions when the server is faulty. Since it was discovered that 
fork-linearizability does not allow for protocols that are wait-free in all executions where the server is 
correct [2 ], the weaker condition of fork sequential consistency was expected to be a promising direction 
to remedy this shortcoming ||2j[TTJ. In this paper we proved that this is not the case, and in fact, fork 
sequential consistency suffers from the same limitation. 



References 

[1] H. Attiya and J. L. Welch. Sequential consistency versus linearizability. ACM Transactions on Computer 
Systems, 12(2):91-122, 1994. 

[2] C. Cachin, A. Shelat, and A. Shraer. Efficient fork-linearizable access to unttusted shared memory. In Proc. 
26 st ACM Symposium on Principles of Distributed Computing (PODC), pages 129-138, 2007. 



[3] Collabnet, Inc. Subversion project, http://subversion.tigris.org/ Last accessed Apr. 2008. 

[4] Google, Inc. Google Docs. |http : / / docs . google . com/ Last accessed Apr. 2008. 

[5] M. Herlihy. Wait-free synchronization. ACM Transactions on Programming Languages and Systems, 
11(1): 124-149, Jan. 1991. 

[6] M. P. Herlihy and J. M. Wing. Linearizability: A correctness condition for concurrent objects. ACM 
Transactions on Programming Languages and Systems, 12(3):463^-92, July 1990. 

[7] L. Lamport. How to make a multiprocessor computer that correctly executes multiprocess programs. IEEE 
Transactions on Computers, 28(9):690-691, 1979. 



6 



[8] J. Li, M. Rrohn, D. Mazieres, and D. Shasha. Secure untmsted data repository (SUNDR). In Proc. 6th 
Symp. on Operating Systems Design and Implementation ( OSDI), pages 121-136, 2004. 

[9] N. A. Lynch. Distributed Algorithms. Morgan Kaufmann, San Francisco, 1996. 

[10] D. Mazieres and D. Shasha. Building secure file systems out of Byzantine storage. In Proc. 21st ACM 
Symposium on Principles of Distributed Computing (PODC), pages 108-1 17, 2002. 

[11] A. Oprea and M. K. Reiter. On consistency of encrypted files. In Proc. 20th Intl. Symp. on Distributed 
Computing (DISC), volume 4167 of Lecture Notes in Computer Science, pages 254-268, 2006. 

[12] M. Pease, R. Shostak, and L. Lamport. Reaching agreement in the presence of faults. Journal of the ACM, 
27(2):228-234, Apr. 1980. 

[13] Wikipedia. List of file systems, distributed file systems section. | http : / /en . wikipedia . org/wiki/| 
|List_of _f ile-systems#Distributed_f ile_systems] Last accessed Apr. 2008. 

[14] J. Yang, H. Wang, N. GU, Y. Liu, C. Wang, and Q. Zhang. Lock-free consistency control for web 2.0 
applications. In Proc. 17th Intl. Conference on World Wide Web (WWW), 2008. 



7 



